
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 1 3- 1 450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/818,123 


03/27/2001 


Frank Sauer 


2001P05535US 


8633 



7590 01/28/2005 

Siemens Corporation 
Intellectual Property Department 
186 Wood Avenue South 
Iselin, NJ 08830 



EXAMINER 



GOOD JOHNSON, MOTILEWA 



ART UNIT 



PAPER NUMBER 



2672 

DATE MAILED: 01/28/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



v/l/vtr MlsllUfl OUIilUiCii y 


Application No. 

09/818,123 


Applicant(s) 

SAUER, FRANK 


Examiner 

Motilewa A. Good-Johnson 


Art Unit 

2672 




- The MAILING DATE of this communication appears on the cover sheet with the correspondence ac 


idress 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 27 September 2004 . 
2a)^ This action is FINAL. 2b)Q This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 8-14 and 17-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) £3 Claim(s) 8-14 and 17-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
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DETAILED ACTION 

1 . This office action is responsive to the following communications: Application, 
filed 03/27/2001; Amendment A, filed 06/30/2003; Amendment B, filed 12/05/2003; 
Amendment C, filed 03/24/2004. 

This action is made final. 

2. Claims 8-14 and 17-30 are pending in this application. Claims 8, 19, 23 and 27 
are independent claims. 

3. The present title of this application is "Augmented Reality Guided Instrument 
Positioning with Depth Determining Graphics" (as originally filed). 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 8-14 and 17-30 rejected under 35 U.S.C. 103(a) as being unpatentable 
over Fukunaga, U.S. Patent Number 6,346,940, "Virtualized Endoscope System", 
02/12/2002, filed 02/27/1998, in view of Manwaring et al., U.S. Patent Number 
5,638,819, "Method and Apparatus for Guiding an Instrument to a Target", 06/17/1997. 
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Regarding claim 8, Fukunaga discloses an apparatus for augmented reality 
guided instrument positioning, comprising: a graphics guide generator (col. 7, line 65 - 
col. 8 f line 10) for generating a first graphic target. 

Fukunaga further discloses (col. 2, lines 45-58, preparing a shape of an image 
with a hidden surface) (col. 2, lines 64-65, a guiding marker preparation unit displaying 
guiding markers) (col. 2, lines 66-67, guiding markers used to guide the insertion of an 
actual or virtual endoscope) (col. 2, lines 59-63, a compass image preparation unit 
which generates a compass image over a first image) and further discloses in col. 2, 
lines 1-7, even if the position is known the depths are difficult to discern and a user 
would have to estimate the depth of a cavity before advancing an endoscope tip. 

However, it is noted that Fukunaga fails to disclose depth marker indicating a first 
target depth of a hidden target, and a graphic instrument depth marker indicating a 
depth of an instrument; and a rendering module for rendering a virtual view of the first 
graphic target depth marker overlaid on an actual view of an object including the target, 
and rendering a virtual view of the graphic instrument depth marker overlaid on the 
actual view such that a proximity of a predetermined portion of the instrument to the 
target is ascertainable based on a position of the graphic instrument marker with 
respect to the first graphic target depth marker. 

Manwaring discloses position data that informs the user of the instrument in 
relation to X,Y,Z coordinate system, Z being depth, of the distance between the 
instrument and a target, col. 5, lines 40-45, and knowing at any instant the location of 
the instrument to the alignment feature, col. 6, lines 23-35. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to include the depth information, as disclosed in Manwaring, of the 
instrument, alignment and target, as a marker, in the guiding marker preparation unit 
disclosed in Fukunaga, overlaid on the view of the target to ascertain the position of the 
instrument with respect to the target depth marker, to allow an operator to quickly and 
easily obtain guiding images for guiding an instrument through the actual cavity. 

Regarding claim 9, Fukunaga discloses graphics proximity marker generator 
determines an optimal location for the predetermined portion of the instrument with 
respect to the target and calculates the proximity of the predetermined portion of the 
instrument to the target based on the optimal location, (col. 2, lines 53-58, preparation 
unit displays a focus mark, which Examiner interprets as a graphics proximity marker, to 
mark a condition inclusive of the location of the condition in the object , because the 
mark is inclusive of the location, Examiner interprets the focus mark as being proximate 
to the predetermined portion of the instrument based on an optimal location) 

Regarding claim 10, it is noted that Fukunaga fails to disclose the graphics guide 
generator determines a second graphic target depth marker indicating a second target 
depth of the target, rendered as a virtual view overlaid on the actual view by the 
rendering module, wherein a distance between the virtual view of the first graphic target 
depth marker and the virtual view of the second graphic target depth marker is a 
predetermined target depth range of the target. 

Manwaring discloses slices providing a reference to maintain a correct sense of 
orientation, col. 5, lines 14-16, and position data that informs the user of the instrument 
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in relation to X,Y,Z coordinate system, Z being depth, of the distance between the 
instrument and a target, col. 5, lines 40-45, and knowing at any instant the location of 
the instrument to the alignment feature, col. 6, lines 23-35. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to include second depth information, as disclosed in Manwaring, of 
the instrument, alignment and target, as a marker, in the guiding marker preparation unit 
disclosed in Fukunaga, overlaid on the view of the target to ascertain the position of 
each surface slice in the target, to allow an operator to quickly and easily obtain guiding 
images for guiding an instrument through the actual cavity. 

Regarding claim 1 1 , Fukunaga discloses the proximity corresponds to a final 
forward position of the predetermined portion of the instrument with respect to the 
target, (col. 8, lines 1-10) 

Regarding claim 12, it is noted that Fukunaga fails to discloses graphics guide 
generator generates a third graphic target depth marker, rendered as a virtual view 
overlaid on the actual view by the rendering module indicating an outer surface of the 
target, wherein the virtual view of the first graphic target depth marker indicates an inner 
portion of the target. 

Manwaring discloses slices providing a reference to maintain a correct sense of 
orientation, col. 5, lines 14-16, and position data that informs the user of the instrument 
in relation to X,Y,Z coordinate system, Z being depth, of the distance between the 
instrument and a target, col. 5, lines 40-45, and knowing at any instant the location of 
the instrument to the alignment feature, col. 6, lines 23-35. 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to include second depth information, as disclosed in Manwaring, of 
the instrument, alignment and target, as a marker, in the guiding marker preparation unit 
disclosed in Fukunaga, overlaid on the view of the target to ascertain the position of 
each surface slice in the target, to allow an operator to quickly and easily obtain guiding 
images for guiding an instrument through the actual cavity. 

Regarding claim 13, proximity comprises a first measure of proximity for 
indicating a front portion of the target and a second measure of proximity for indicating a 
back portion of the target, the front portion corresponding to entering the target and the 
back portion corresponding to exiting the target, (col. 8, lines 1-10, markers indicate the 
path to take, which Examiner interprets as including a front to back and back to front 
proximity) 

Regarding claim 14, the graphic guide generator determines at least one 
graphics path marker that identifies at least one path for the instrument to the target; 
wherein said rendering module renders a virtual view of the at least one graphics path 
marker as an overlay on the actual view to which a portion of the instrument visible in 
the actual view is alignable. (col. 8, lines 1-10, markers indicate the path to take, and 
the type of guiding marker, and further discloses based on the viewpoint position and 
view direction the computation unit determines the position of the instrument for relative 
to an outer shape, view direction and reference direction, col. 13, lines 30-54, which 
Examiner interprets as an alignable feature) 
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Regarding claim 1 7, a graphic guide generator for determining at least one 
graphics path marker that identifies a path to the target, wherein the graphics path 
marker is a shape comprising a window, wherein the window marks the path. (col. 8, 
lines 1-10, markers indicate the path to take, and the type of guiding marker) 

Regarding claim 18, a graphic guide generator for determining at least one 
graphics path marker that identifies a path to the target, wherein the graphics path 
marker is a pattern of lines centered on the target, wherein at least two lines are spaced 
differently from at least one other line, wherein the at least two lines frame the path, 
(col. 7, lines 65- col. 8, lines 10) 

Regarding claims 19, 23 and 27, they are rejected based upon similar rational as 
above claim 8. 

Regarding claims 20, 24, and 28, they are rejected based upon similar rational as 
above claim 10. 

Regarding claims 21, 25, and 29, they are rejected based upon similar rational as 
above claim 10. 

Regarding claims 22, 26 and 30, they are rejected based upon similar rational as 
above claim 14. 

Conclusion 

6. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 



Response to Arguments 

7. Applicant's arguments filed 09/27/2004 have been fully considered but they are 
not persuasive. 

Applicant argues that Fukunaga fails to disclose or suggest overlaying a virtual 
image on an actual view. Fukunaga discloses an image processing system and real 
images displayed on the monitor, col. 5, lines 30-52. Fukunaga further discloses 
preparing an endoscopic image, which is an image of a three dimensional model which 
is a virtual representation of the subject, col. 6, lines 15-34. It is therefore the 
interpretation of the Examiner that Fukunaga discloses overlaying a virtual image, col. 
6, lines 44-56, on an actual view, i.e. the three dimensional model, because the three- 
dimensional model is representative of the viewpoint and view direction represented by 
an endoscopic image. 
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Applicant argues that Manwaring fails to disclose or suggest rendering a virtual 
view of the first graphic target depth marker overlaid on an actual view of an object. 
Fukunaga discloses guiding markers used to guide the insertion of an actual or virtual 
endoscope col. 2, lines 66-67, a compass image preparation unit which generates a 
compass image over, i.e. overlaying, a first image col. 2, lines 59-63, and further 
discloses in col. 2, lines 1-7, even if the position is known the depths are difficult to 
discern and a user would have to estimate the depth of a cavity before advancing an 
endoscope tip, which Examiner interprets as suggesting a virtual view of a graphic 
target marker overlaid on an actual view. Manwaring discloses position data that 
informs the user of the instrument in relation to X, Y, Z coordinate system, Z being 
depth, of the distance between the instrument and a target, col. 5, lines 40-45, and 
knowing at any instant the location of the instrument to the alignment feature, col. 6, 
lines 23-35, which Examiner interprets as providing motivation to include depth marker 
overlaid on an actual or virtual view of an object. 

Applicant argues that Fukunaga fails to disclose rendering or a rendering module 
for a rendering a virtual view of a path marker. Fukunaga discloses a computation unit 
and a preparation unit, in which the computation unit is used to prepare for display an 
indicator image showing the state of the virtual endoscope, see figure 4. Examiner 
interprets the computation unit as the rendering module, which generates the virtual 
image of the actual image, i.e. indicator image, showing a virtual endoscope view 
position, direction and depth marker, col. 7, lines 3-39. 



Application/Control Number: 09/818,123 Page 10 

Art Unit: 2672 

Lastly, Applicant argues that Manwaring fails to disclose a portion of the 
instrument visible in the actual view and the overlaid in the actual view. Fukunaga 
discloses guiding markers used to guide the insertion of an actual or virtual endoscope 
col. 2, lines 66-67. Manwaring discloses position data that informs the user of the 
instrument in relation to X, Y, Z coordinate system, Z being depth, of the distance 
between the instrument and a target, col. 5, lines 40-45, and knowing at any instant the 
location of the instrument to the alignment feature, col. 6, lines 23-35. Therefore it is the 
Examiner's interpretation that it would have been obvious to include the overlay of the 
instrument and depth-guiding marker to know at any instant the location of the 
instrument to the alignment feature. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Motilewa A. Good-Johnson whose telephone number is 
(703) 305-3939. The examiner can normally be reached on Monday, Tuesday and 
Thursday 9:00 AM - 6:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone number for 
the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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Examiner 
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